Storage Access Characteristics of Computational Science Applications

Computational science applications are driving a demand for increasingly powerful storage systems. While many techniques are available for capturing the I/O behavior of individual application trial runs and specific components of the storage system, continuous characterization of a production system remains a daunting challenge for systems with hundreds of thousands of compute cores and multiple petabytes of storage. As a result, these storage systems are often designed without a clear understanding of the diverse computational science workloads they will support. In this study, we outline a holistic methodology for scalable, systemwide I/O characterization that combines storage device instrumentation and static file system analysis with a new mechanism for capturing detailed, application-level behavior. We demonstrate the effectiveness of our methodology by performing a multilevel, two-month study of Intrepid, a 557-teraflop IBM Blue Gene/P system. During that time, we captured applicationlevel I/O characterizations from 6,481 unique jobs spanning 38 science and engineering projects with up to 163,840 processes per job. We also captured patterns of I/O activity in over 8 petabytes of block device traffic and summarized the contents of file systems containing over 191 million files. From this collection of data we are able to quantify systemwide trends such as how application behavior changes with job size, the “burstiness” of the storage system, and the change in file system contents over time. We also identify the top ten storage users by application domain and investigate how their I/O strategies relate to I/O performance. One of these applications is then selected as a case study in I/O tuning based on integrated I/O characterization. We then use the results of our study to highlight trends that will affect the design of future storage systems, and we identify opportunities for improvement in I/O characterization methodology.